Intelligent user mode selection in an eye-tracking system

ABSTRACT

A personal computer system comprises a visual display, an imaging device adapted to provide eye-tracking data by imaging at least one eye of a viewer of the visual display, and identifying means for recognizing the viewer with reference to one of a plurality of predefined personal profiles. The personal computer system further comprises an eye-tracking processor for processing the eye-tracking data. According to the invention, the eye-tracking processor is selectively operable in one of a plurality of personalized active sub-modes associated with said personal profiles. The sub-modes may differ with regard to eye-tracking related or power-management related settings. Further, the identifying means may sense an identified viewer&#39;s actual viewing condition (e.g., use of viewing aids or wearing of garments), wherein the imaging device is further operable in a sub profile mode associated with the determined actual viewing condition.

TECHNICAL FIELD OF THE INVENTION

The invention disclosed herein generally relates to eye tracking(determination of eye position, gaze point or gaze angle) for providinginput data to a computer system. In particular, the invention providesan energy-efficient implementation of eye tracking assisted by anartificial light source adapted to determine the gaze point of an eyewatching a visual display forming part of a portable or stationarypersonal computer system, a TV, a heads-up display in a vehicle, anear-eye display or a display in a communication device with imaging andcomputing capabilities, such as a mobile telephone.

BACKGROUND OF THE INVENTION

Monitoring or tracking eye movements and detecting a person's gaze pointcan be used in many different contexts. Eye tracking data can be animportant information source in analysing the behaviour or consciousnessof the person. It can be used both for evaluating the object at whichthe person is looking and for evaluating the respective person. Thediverse uses of gaze point detection include studies on the usability ofsoftware and different types of interfaces; evaluation of web pages,advertising and advertisements; provision of means for educating pilotsin simulator environments and for training surveillance personnel insecurity-critical roles; and research in psychology, behaviouralsciences and human perception. A field which has attracted an increasinginterest in recent years is the evaluation of advertising and othermarketing channels.

Eye-tracking techniques can also be used in a human-machine interface(HMI): a user can control a computer by just looking at it. Such eyecontrol can be applied as sole interaction technique or combined withkeyboard, mouse, physical buttons and voice. Eye control is used incommunication devices for disabled persons and in various industrial andmedical applications.

While eye-tracking systems are utilized in a growing range ofapplications aimed at professionals, they are rarely included asstandard peripherals in or as integral parts of new laptops, desktops,smart phones and other personal computer systems. In the case ofbattery-powered systems, concerns that eye-tracking functionalitiesmight impair an otherwise optimized energy management may be one reasonfor this absence.

US 2005/199783 A1 describes a technique for switching a generic devicebetween a power-up state, a sleep state and a power-off state on thebasis of eye detection data relating to a user. Only the presence orabsence of an eye is of concern, not the gaze angle. A detected presenceof an eye causes switching from the sleep state to the power-up state,while a detected absence causes switching down from the power-up stateto the sleep state, and then to the power-off state of the device. Whilethis document describes how an eye detection result, namely eyepresence, can be used to improve the energy performance of the genericdevice, it does not address the power management problem in the eyedetection equipment itself. Nor does it propose any solution that istailored to, and benefits from, the particularities associated with eyedetection activity.

Similarly, U.S. Pat. No. 5,835,083 A and WO 2008/056274 A1 discuss howgaze-point measurements can be used to control a power state of a visualdisplay, so that power consumption is reduced when a user's eyes andhence a user's attention are not directed to the device. They also donot address power management in the eye tracker itself.

US 2010/079508 A1 discloses an electronic device with a display, such asa handheld computer, which is able to detect whether a user's gaze isdirected towards the display. The device implements a power managementscheme which involves the device operating in multiple modes to conservepower and utilizing gaze detection operations to assist in determiningan appropriate mode in which to operate. The different modes of thedevice involve adjusting the brightness of the display or turning on/offcomponents such as a display or a touch screen in order to conservepower. Additionally, certain aspects of the gaze detection capabilitiesof the device can be adjusted to conserve power depending on the mode inwhich the device is operating.

A further possible reason that eye-tracking functionalities are not yetincluded as standard components in personal computer systems is thehassle that may arise in connection with transitions between differentusers. Eye-tracking systems that attempt to offer some relief in thisrespect have emerged, e.g., the one disclosed in U.S. Pat. No. 7,657,062B2, which is configured to match an estimated time distribution of theazimuth gaze direction against a predetermined distribution of the samequantity measured under laboratory conditions. The gaze direction iseither measured with respect to a fixed coordinate system or to a systemfixed about a human head.

SUMMARY OF THE INVENTION

In view of the above concerns, it is an object of the present inventionto propose a personal computer system with improved power managementfunctionalities in respect of eye-tracking equipment included therein.It is a particular object to improve power management in a system ofthis type while preserving low latency in respect of user interactionsat all instants when the device is operating. Yet another object is toprovide an eye-tracking system that can be integrated in a personalcomputer system (e.g., desktop or laptop computer, tablet computer,notebook, net book, TV, smart phone, personal digital assistant, digitalcamera, heads-up display, near-eye display) without burdening the energyperformance of the computer system. A further object is to facilitatethe use of one eye-tracking enabled personal computer system by manyusers. In such a system, it is a particular desire to simplify andaccelerate the transitions between different viewers and/or decrease therisk of human errors in connection with the transitions.

At least one of these objects is achieved by a method, computer programproduct, and personal computer system, as set forth in the independentclaims. The dependent claims define embodiments of the invention.

A personal computer system includes a visual display, an imaging devicefor providing eye-tracking data by imaging a portion of the face(preferably including at least one eye) of a viewer of the visualdisplay, and further one or more input means for accepting eye-trackingcontrol data and other input data. The imaging device may include acamera and an optional light source for illuminating an eye in anon-axis or off-axis fashion, or for producing at least one cornealreflection (or glint, or first Purkinje reflection) to facilitate eyetracking. Such illumination which the imaging device provides inaddition to natural or background light sources will be referred to asactive illumination. The other input data may include pointing devicesignals, keyboard characters, keyboard combinations, visual data otherthan eye-tracking data, proximity sensing, data acquired by an acoustictransducer and the like.

According to a first aspect of the invention, the imaging device isoperable in at least an active mode, a ready mode and an idle mode. Inthe active mode, the imaging device is fully operable as regardsaccuracy, detection range and other performance parameters that mayinfluence the momentary power consumption of the device. The ready modeand the idle mode represent power-saving alternatives to the activemode, which differ at least with respect to their respective wake-uptimes. More precisely, the wake-up time required to switch from readymode into active mode is shorter than the wake-up time required toswitch from idle mode into active mode.

The invention achieves at least one of its objects since the proposedenergy management technique takes into account the fact thateye-tracking algorithms generally contain recursive filtering (e.g.,Kálmán filtering), wherein the accuracy is improved gradually with thenumber of iterations, or are dependent on previous measurements,intermediate data or partially processed data to be used as initialguesses for subsequent tracking. An eye-tracking algorithm of this typedoes not provide accurate and complete measurement data from the momentit is cold-started, but only after a wake-up time period has elapsed.Hence, in the prior art, the requirements of low energy consumption andresponsiveness (low user latency) are clearly conflicting. The inventionalleviates this difficulty by proposing a ready mode, in which theeye-tracking equipment operates at lower but non-zero power, so that aportion of the previous measurements, intermediate data or partiallyprocessed data remain updated and available to support and facilitatesubsequent measurements when the equipment switches back into its activemode.

The active, ready and idle mode may differ regarding the state of(components of) the imaging device only, but may also differ withrespect to operational parameters of other components in the personalcomputer system. For instance, the fact that the imaging device entersits off mode may trigger turn-off of a backlight in the visual display.

The imaging device may consist of a camera only, preferably a digitalcamera, but may also include further components, such as a light sourcefor assisting the camera, e.g., by emitting non-visible light pulsespreferably in the infrared or near-infrared range. Within the imagingdevice, therefore, either the camera only, the light source only or acombination of these and possible further components may behavedifferently in the active, ready and idle mode, respectively. As usedherein, the term imaging device is not restricted to optical cameras,but is also intended to cover acoustic (e.g., ultrasound),electromagnetic (e.g., radar) sensors. The term also extends beyondthose sensors which produce images that are perceived as such by a humanviewer, thereby covering sensors formed as arrangements of asingle-digit number of pixels, sensors including highly distortingpre-lenses intended to favour optical accuracy in regions of interestover other regions etc. Furthermore, the imaging device may be directedtowards one or more of the viewer's eyes only but may as well image alarger portion of the face so as to determine a relative head pose, andthe gaze may be determined based on the position of at least one eye inthe face.

The active and ready modes may differ with regard to the data theimaging device provides. In the active mode, both eye position and eyeorientation may be provided. (To make this statement precise, theimaging device may output processed data representing eye position andeye orientation or, if it lacks appropriate processing capabilities ofits own, the imaging device may output sufficient raw image data that aprocessor is able to determine eye position and eye orientation. Thereceiver of the output data may be a processor responsible for executinga graphical user interface forming part of application software, adesktop environment or the like.) In the ready mode, however, either ofthese may be omitted to save resources, preferably eye orientation sothat only eye position is provided. The eye position tracking mayproceed throughout the ready mode, though preferably at a lower framerate than in the active mode, so that up-to-date information on the eyeposition is readily at hand at the moment the imaging device switchesback into active mode. This reduces the wake-up time significantly,while the energy consumption in ready mode may be limited significantly.

Alternatively, the active and ready modes may differ with respect to thenumber of distinct tracked eye features (e.g., pupil location, cornealreflections) on which the imaging device bases the eye-tracking data. Inactive mode, the eye tracking may be based on two or more features. Forexample, the eye-tracking processing may be based on reflections of noless than two distinct light sources (including the case where thereflections are captured within different camera frames), whereas inready mode, the eye tracking may be based on a single distinctreflection, such as may be obtained using one single light source(including the case where a reflection of this light source is imaged atmultiple points in time within different frames), that is, theeye-tracking processing is able to complete based on data from a singlereflection. It is recalled that gaze tracking according to thepupil-centre-corneal-reflection (PCCR) approach requires as input thelocations of a pupil and a corneal reflection that are simultaneous ornear-simultaneous (see, e.g., the paper General Theory of Remote GazeEstimation Using the Pupil Center and Corneal Reflections by E. D.Guestrin and M. Eizenmann, IEEE Transactions on Biomedical Engineering,vol. 53, no. 6, pp. 1124-1133 (June 2006), included herein byreference). The camera-to-eye distance may be a further input datasource in PCCR gaze tracking. It is described in U.S. Pat. No. 7,572,008how this distance can be estimated on the basis of two distinct cornealreflections. Accordingly, the eye tracker may refrain from updating thelatest estimate of the camera-to-eye distance when in the ready mode butmay do so intermittently in the active mode.

Further advantageous examples indicating how the active and ready modescan be configured in detail are noted in Table 1.

TABLE 1 Mode configurations Active mode Ready mode The imaging devicetracks a pupil The imaging device tracks a pupil location and at leastone corneal re- location. flection. The imaging device applies activeThe imaging device does not apply illumination, which enables activeillumination. tracking of a corneal reflection. One light source and onecamera Two light sources and one camera are active. are active. Onelight source and one camera One light source and two cameras are active.are active. One camera is active. One light source and one camera areactive. The imaging device operates at full The resolution of theimaging device resolution. is reduced by binning pixel groups, e.g., by2 × 2 (ratio 4:1), 4 × 4 (ratio 16:1), 1 × 2 (ratio 2:1), 2 × 1 (ratio2:1), wherein multiple pixels are read out as one. Preferably, sincebinning in- creases the sensitivity of the imaging device, an associatedlight source is operated at lower intensity or is turned off completely.Additionally, the exposure time of the imaging de- vice may beincreased, thereby fur- ther increasing sensitivity at the cost of someaccuracy. The imaging device operates at a The imaging device operatesat a relatively lower binning ratio, e.g., relatively higher binningratio, e.g., 2:1. 16:1. The imaging device measures or The imagingdevice measures or es- estimates an eye position in world timates an eyeposition in image- coordinates (e.g., n-dimensional plane coordinates(e.g., (n − 1)- coordinates or 3-dimensional dimensional coordinates or2- coordinates). dimensional coordinates).It is pointed out that the scope of the invention includes combinationsof the above pairs as well. Likewise, binning may refer to analoguebinning, such as by reading out pixel charges in a group-wise fashion,so that luminous energy received at a plurality of pixels contribute toone value. It may also refer to digital binning in the sensor, which mayform part of a pre-processing step involving adding or combiningread-out data pixel values in processing hardware.

Moreover, in a system where plural cameras and/or plural light sourcesare provided, the ready mode may involve using a smaller number of thesedevices. Since estimations based on a smaller data set may have greaterstatistical variance, this mode may lead to slower and less accurate eyetracking data but may still provide sufficient information tosignificantly shorten the time for switching into active mode andcollecting relevant eye-tracking data in comparison with a cold startfrom idle mode.

The input means in the personal computer system may consist of adedicated input means on the one hand and general-purpose input means onthe other. It may also consist only of either of these, as mentioned inthe next paragraph. The dedicated input means are used to inputeye-tracking control data only, whereas the general input means acceptall other input data than eye-tracking data, that is eye-trackingcontrol data and other input data. Because the dedicated input means isused only for eye-tracking control data, the operating system mayallocate to it abilities to activate the eye tracker with lower delaythan the general-purpose input means would achieve. The dedicated inputmeans may be configured as a camera for detecting predefined facegestures, predefined body gestures or a microphone for detecting apredefined voice pattern. Advantageously, the camera used for thispurpose is identical to the at least one imaging device that suppliesthe eye-tracking data. The dedicated input means may further be embodiedas a hardware or software button, an IR sensor, a motion sensor, aproximity sensor, a touch-sensitive layer of a visual display or aportion thereof. In the latter case, one touch-sensitive display maycomprise both an area acting as a dedicated input means and an areaacting as a general-purpose input means.

Said eye-tracking control data entered via the dedicated input means maybe an activation click, that is, a mouse-click-type signal supplementinga gaze point on the visual display to achieve a similar interface asthat offered by a conventional pointing device, although this need notbe organized on the basis of the pointer location as such conventionalsystems generally are. A completely hands-free HMI, in which all inputdata are entered either in the form of eye-tracking data or eye-trackingcontrol data, is envisioned. Additional input means in such hands-freeHMI may include acoustic, haptic or optic transducers and the like butis devoid of devices adapted to be mechanically manipulated usingfingers, hands or other body parts.

Said eye-tracking control data may also be used to switch theeye-tracking functionalities between an enabled state and a disabledstate, which may be particularly attractive for users conscious aboutpersonal integrity. As one possible option, the dedicated control meansmay be configured to force the imaging device into idle mode.

Alternatively, the dedicated input means may trigger an interrupt bywhich the imaging device is forced into active mode. The triggering maybe achieved by functionally connecting the dedicated input means to aninterrupt means (e.g., an interrupt pin) provided on the imaging deviceor on a processor associated therewith. Preferably, the dedicated inputmeans is functionally disconnected from the interrupt means in theactive mode, so as not to perturb the work of the imaging device duringactive mode, wherein the computational load is relatively higher than inother modes. By using an interrupt in this manner, the total latencyassociated with a switching into the active mode is reduced incomparison with the case of triggering the switching by means of thegeneral-purpose input means, which typically have an inherent latency.Most of today's low-grade and middle-grade keyboards, mice, touchscreens and other general-purpose I/O devices, of the type which a usermay be expected to connect to a personal computer system in theirpossession, operate by line scanning followed by interrupt generation.Such an interrupt is generated indirectly, not by the user's actuationbut by the scanning result. This principle of operation incurs a delay,which is typically negligible in the intended use of the I/O device(e.g., typing) and therefore rarely improved on by the manufacturer, butwhich makes a general-purpose I/O device poorly fit to inputeye-tracking control data. Indeed, the latency contributed by the I/Odevice adds to the wake-up time of the imaging device itself, so thatthe total latency may become larger than is acceptable in a givenapplication. This embodiment of the invention resolves the problem bytriggering an interrupt directly.

In a further embodiment, the imaging device is powered separately, suchas via an autonomously controllable electric switch connecting it to adrive power necessary for its operation. With this setup, the idle modemay consist in a complete power-off state of the imaging device. Hence,advantageously, the dedicated input means forces the imaging device intoidle mode by disconnecting it from said drive power.

The active, ready and idle mode may differ with respect to an operatingfrequency of the imaging device. Generally, the operating frequency mayrefer to any frequency characterising a component within the imagingdevice, to the extent that the frequency influences the momentary powerconsumption. In particular, the operating frequency may be the samplingfrequency (or frame rate) of a camera within the imaging means. It mayalso refer to a light-pulse frequency of a pulsed light source used inconnection with a camera of this type, wherein each light pulse issynchronized with a sampling instant of the camera. In particular, theactive and the ready mode may differ in terms of the operatingfrequency, wherein the ready mode is associated with a lower, non-zerofrequency which maintains eye-tracking at a less accurate level. Such aless accurate level is yet configured with the aim of promoting fastswitching from the ready mode into the active mode.

As a further option, which is particularly advantageous in connectionwith an eye tracker that utilizes active illumination, the operation ofthe imaging device in ready mode may include reducing an illuminationintensity of the light source from the value it has in active mode. Theillumination may even be dispensed with altogether, by turning the lightsource off, wherein the camera may optionally operate with longerexposure duration and/or pixel binning, so that the imaging device stillprovides output data although at a relatively lower quality. While theillumination is turned off, the duties normally fulfilled by the cameramay alternatively be carried out by a camera for non-visible light, suchas a camera sensitive to infrared radiation in or around the wavelengthrange corresponding to human body temperature.

The personal computer system may include a viewer presence detector,which is adapted to produce a positive and/or a negative detectionsignal causing the imaging device to transfer between modes accordingly.The presence detector may be a proximity detector or motion detectoroperating on the basis of, e.g., optic, acoustic, electromagnetic orcapacitive measurements. It is noted that the presence detection mayrelate either to proximity of a viewer's eye to the imaging device or toproximity of the viewer's face, head or body to the imaging device orthe personal computer system.

It is particularly advantageous to embody the viewer presence detectoras a sensor arranged to detect proximity of a viewer's finger (or hand)to a button, scroll wheel or other hardware that is typically used forinputting data during a work session. The proximity sensor may forexample be mounted in a push button acting as a dedicated input means inthe sense above, notably for entering activation clicks with referenceto a visible item appearing at the gaze position on a display. Such anactivation click may cause activation of the item in the same manner asa conventional mouse click does. When the viewer has been detected inthe above manner as being present, it is ensured that the imaging deviceenters ready mode, so that a switching to active mode, in case work isresumed, can be performed in very short time. The switching time may befurther reduced if this embodiment is used in conjunction with otherfeatures of this invention, such as by using a direct interrupt to carryout this mode switching.

Alternatively or additionally, the personal computer may include anidentifying means for determining the identity of a current viewer. Theidentification may be carried out with reference to a set of predefinedpersonal profiles, wherein each is associated with personalized activemodes including, e.g., values of parameters relevant to eye-trackingand/or energy management. Parameters relevant to eye-tracking mayinclude eye-tracking calibration settings such as:

-   -   any viewer-dependent settings necessary for executing an        eye-tracking algorithm or more particularly a gaze-tracking        algorithm within the PCCR approach (see above), such as        curvature and size of the cornea, position of the fovea,        measured bright-pupil response and the like;    -   pre-stored calibration results, e.g., from a preliminary        training session where the viewer is requested to gaze at        predetermined display locations; and    -   information indicative of the appearance of the eye area of the        viewer, such as sclera colour, iris colour, complexion, presence        of coloured eyeglasses and the like.        Parameters relevant to energy management may include settings        indicating the typical length of a work session, a delay to be        applied before a power-saving mode is entered, a desired image        resolution or sampling frequency. The values may be pre-set by        the viewer or by a system administrator with reference to an        existing person. Alternatively, they may be generic in nature        and pre-stored by a system designer to suit different categories        of users.

As a further development of the preceding embodiment, the identifyingmeans is an imaging device which is capable of sensing a viewer's (or inparticular, an identified viewer's) actual viewing condition.Preferably, the actual viewing condition is partially based on visualdata or entirely based on visual data. By an actual viewing condition isunderstood the presence of viewing aids, such as eyeglasses or contactlenses, or the wearing of certain garments, such as a cap or a veil,which information may improve or render more economical the acquiringand/or computational processing of eye-tracking data. A key to detectingthe presence of contact lenses is to record occasions when a cornealreflection (glint) slips out into the sclera due to an oblique gazeangle; the average gaze angle at which the reflection leaves the corneamay be expected to be smaller for an eye wearing a contact lens than fora naked eye. Such adaptations may include modifying eye illumination,controlling optical filtering or compensating reflections and/orgeometrical deformations produced by refractive elements in proximity ofthe eye. As an example of a potential energy saving made possible byknowing the actual viewing condition, the imaging device may be operatedat lower resolution and/or without active illumination if the viewer'seye region in the actual viewing condition is known to provide goodcontrast in itself. The adaptations may advantageously be encoded as oneor more sub-profiles associated with the personalized profiles discussedabove. For instance, the active mode of the imaging device may bedifferentiated into active modes for persons A, B, C, etc., wherein theactive modes for person A may be further subdivided into sub-profiles“person A without eyeglasses”, “person A wearing clear eyeglasses” and“person A wearing sunglasses”. In particular, one or more sub-profilemodes may comprise settings to be applied in a pre-processing stepcompensating geometric deformations resulting from a refractive viewingaid, wherein different sub-profiles may differ with respect to thevalues of these settings. The pre-processing step may be carried out bythe imaging device or an eye-tracking processor elsewhere in thepersonal computer system which is responsible for deriving eye-trackingoutput data on the basis of the data provided by the imaging device.

It is noted that the sub-profiles modes need not be hierarchicallydefined, as ‘children’ of a personalized sub-mode, but may also berepresented as modes in their own right. In this situation, it may besuitable to encode in the data structures the fact that groups of modesare related insofar as they belong to a particular viewer, so as tofacilitate the subsequent task of switching between the modes. Indeed,this reflects the fact that switching between sub-profiles is typicallymore frequent than switching between persons.

In a second and third aspect, the invention provides a method foroperating a personal computer system including eye-trackingfunctionalities as well as a computer program product for performing themethod by means of a programmable processor communicatively connectedto—or constituting—said personal computer system. The above featureswhich have been outlined within the first aspect readily carry over tothe second and third aspect, in which they may be used to advantage.

In one embodiment, a personal computer system comprises the followingfeatures:

-   -   a visual display;    -   an imaging device adapted to provide eye-tracking data by        imaging at least one eye of a viewer of the visual display;    -   identifying means for recognizing the viewer with reference to        one of a plurality of predefined personal profiles; and    -   an eye-tracking processor for processing the eye-tracking data        provided by the imaging device into processed eye-tracking data.        According to this embodiment, the eye-tracking processor is        selectively operable in one of a plurality of personalized        active sub-modes associated with said personal profiles.

In this embodiment, the identifying means may apply any per se knownbiometric identification technique, such as fingerprint recognition,iris recognition or face recognition. It may as well utilizeeye-tracking specific data, such as eye-blinking pattern, gaze pattern,fixation pattern, saccade pattern, eye geometry, average gazecalibration error and the like. In particular, the identifying means maycomprise an imaging device configured to identify a viewer's iris and/orfacial features. In the interest of simplifying the computer systemstructurally, the identifying means may use the imaging deviceresponsible for eye tracking for the identification; this isadvantageous, as the imaging device is in general not engaged in eyetracking before the viewer has been identified. Alternatively, theidentifying means may be neither eye-tracking specific nor biometric,but may identify the user by means of a password, activation code orsome other kind of security token.

In this embodiment, further, the eye-tracking processor may beintegrated in the imaging device, preferably as a dedicated device,which is advantageous in cases where the eye-tracking capability isprovided as an embedded system. Alternatively, the eye-trackingprocessor may coincide with a processor elsewhere in the personalcomputer system, which processor may have other responsibilities inaddition to processing eye-tracking data. In particular, theeye-tracking processing may run as one of several parallel processes inan operating system executing on the processor. Independently of thelocation and the level of specialization or autonomy of the eye-trackingprocessor, the processing algorithms to be performed may include amode-specific post-processing step, in particular a step includingpersonal calibration data used to calculate the gaze position of theidentified person.

The above embodiment is advantageous, especially in view of some of theobjects of the invention in that it may reduce the time required toswitch between users and/or the time required for the eye tracker toreadapt when one user adjusts his or her way of interacting with the eyetracker during a work session.

In a further development of this embodiment, a second identifying meansmay sense a viewer's actual viewing condition, as outlined above. Basedon the actual viewing condition thus determined, the eye-trackingprocessor is operated in a sub-profile mode associated with this. Forinstance, one or more sub-profile modes may be defined for eachpersonalized active sub-mode. The second identifying means used forsensing an actual viewing condition may be distinct from the identifyingmeans used for recognizing a viewer. This way, the hardware can beoptimized in view of the respective tasks and time multiplexing can beavoided. Alternatively, the identifying means and the second identifyingmeans may coincide. This has the advantage of reducing the number ofparts and possibly reducing the bulkiness or cost of the eye-trackingequipment. As noted above, it is possible to perform both the viewerrecognition and the eye-tracking tasks in one imaging device, andconsequently this device may in some embodiments fulfill a three-foldpurpose, namely to perform viewer recognition, eye tracking and sensingof an actual viewing condition.

Results produced by the identification means and/or the secondidentification means may be updated during operation. This applies bothto results of recognizing a viewer and, if such are available, toresults of sensing an actual viewing condition. The updating may amountto repeating the operation on which the original result was based. If acurrent personalized sub-mode is not the one associated with the bestmatching personal profile, or a current sub-profile is not the oneassociated with the actual viewing condition, it is concluded that theupdated result is different. The eye-tracking processor then beginsoperation in a mode (that is, a personalized sub-mode or a sub-profilethereof, as the case may be) associated with the updated result.Advantageously, an attempt to re-sense the actual viewing condition ismade relatively more frequently than an attempt to recognize the viewer.As one example, a temporary absence of tracked eyes may be interpretedeither as an eye blinking by a current viewer or a change of vieweraltogether; according to the invention, the primary option is toascertain whether there has been a change in actual viewing conditionand the secondary option is to re-identify the viewer. In particular,the updating of the actual viewing condition may proceed continuously.

As a further development of the above embodiments, the personal computersystem may execute a graphical user interface (GUI) including visibleitems appearing on the display. This provides for a further way ofverifying that the correct mode (i.e., personalized sub-mode orsub-profile) is being used, namely, by comparing the viewer's gaze-pointlocation with the actual location of a visible item in the GUI at apoint in time when the viewer is deemed to gaze at this feature, fromwhich a gaze-point deviation can be derived. An advantageous point intime for thus measuring the gaze-point deviation is during an activationclick entered by means of a dedicated input means, as described above,by which action the viewer inputs data using the GUI. Equivalently, thepoint in time at which the viewer activates a visible GUI item by gazingat it for a prolonged duration may be used. In the case of an extendedvisible item, such as a button on a GUI adapted for click actuation, thedeviation may for instance be measured with reference to a centre orcentroid of the item. The identifying means may receive informationindicating the visible item locations directly from the processor orindirectly, via the visual display, if the display post-processes theprocessor output necessary to determine the final locations of theitems.

As a variation hereto, a personal profile or an actual viewing conditionmay include data indicating a historic average deviation (or some otherstatistical measure of the total deviation), which may then be used asan expected calibration error being one of the properties to aid inrecognizing a known user of the eye-tracking enabled system. Bycomparing the current average gaze-point deviation during a worksession, it may be verified or falsified that the eye-tracking processoris operating in the most suitable mode. As an alternative, theidentifying means may be adapted to select the mode which currentlyprovides the smallest average gaze-point deviation. Alternatively, theoccurrence of a large deviation detected in the gaze data, whichindicates that the current profile is not optimal, can trigger anothermeans for identifying the person.

In a further development of the above embodiment, the personal computersystem further includes a mode-transition indicator. The mode-transitionindicator notifies the viewer that the eye-tracking processor is aboutto begin or has begun operation in a freshly selected mode, that is, afreshly selected (notably updated) personalized sub-mode or asub-profile. This provides the viewer or the new viewer, respectively,with feedback to the effect that the system has adapted to a detectedchange of person or a change of actual viewing condition. Thenotification may be visual, such as by changing a symbol appearing onthe visual display. For instance, the appearance of an icon may change.As another example, if the new mode is a sub-profile, an avatarrepresenting the viewer may mimic the change in actual viewing conditionby beginning to wear (not wear) eyeglasses, a cap etc. Furthermore, thenotification may be non-visual, preferably acoustic or tactile, so asnot to obscure other items on the visual display. In particular, when aknown user X begins a new work session, a natural-language message alongthe lines of “Welcome X!” or “Welcome X! I see you're wearingsunglasses.” may be read aloud.

In a particular embodiment, the viewer is recognized by way of faceidentification. To facilitate this, the identifying means is an imagingdevice with an active illumination capability, so that a face to berecognized can be illuminated in a suitable manner regardless of theambient lighting conditions. Such suitable illumination may besubstantially uniform and may attempt to avoid abrupt local illuminationgradients, e.g., sharp shadows. This typically makes the facerecognition faster and more accurate. The active illumination may usevisible or non-visible light. Advantageously, the active illumination isperformed by an illuminator that is otherwise adapted to provide a pupilreflex required for PCCR gaze tracking. This reduces the complexity ofthe eye-tracking equipment.

It is noted that the invention relates to all combinations of features,even if they are recited in mutually different claims.

BRIEF DESCRIPTION OF THE DRAWINGS

By way of example and not limitation, embodiments of the invention willnow be described with reference to the accompanying drawings, on which:

FIG. 1 illustrates an active, ready and idle mode of an eye-trackingequipment and transitions between these;

FIG. 2 illustrates a further development of the setup in FIG. 1, whereinthe active mode is differentiated into personalised modes;

FIG. 3 is a generalized block diagram of a personal computer system inaccordance with an embodiment of the invention;

FIG. 4 illustrates a set of sub-profile modes of an eye-trackingprocessor in a personal computer system, which modes are subordinate toan active sub-mode of the same component; and

FIG. 5 illustrates a set of active sub-modes of an eye-trackingprocessor in a personal computer system.

All the figures are schematic and generally only show parts which arenecessary in order to elucidate the invention, whereas other parts maybe omitted or merely suggested.

DETAILED DESCRIPTION OF EMBODIMENTS

FIG. 1 schematically shows an active mode A, a ready mode R and an idlemode I of a imaging device in an eye tracker. As outlined above, the eyetracker performs computations of a recursive nature or uses historicdata, so that, on the one hand, a given computing or measuring task maybe facilitated or expedited by results or intermediate results fromprevious computations and measurements and, on the other hand, the eyetracker has considerable wake-up time before it provides accurate andcomplete output data. No result history is maintained in the idle mode,whereas a full result history—allowing the eye tracker to take fulladvantage of the previous computations and measurements—is maintained inthe active mode. The full result history may refer to a moving timewindow of values, whereby the least recent values are discarded as newones are entered. The ready mode is characterized by producing andmaintaining a partial result history (e.g., lower sample rate, fewersamples or samples acquired at lower resolution or lower accuracy),representing a trade-off between low energy consumption and highresponsiveness (quantified as, e.g., a short wake-up time). Likewise, inimplementations where various hardware components are associated with awarm-up or initialization time until the component is in a fullyoperational state, the ready mode may correspond to a partiallyoperational mode, wherein not all components operate at full powerand/or some components are completely disabled. Preferably, the disabledcomponents are those which contribute most significantly to the totalpower consumption and/or have the shortest initialization time

FIG. 3 shows a personal computer system 300, which includes a visualdisplay 310 for displaying output data. The visual display 310 mayproduce an image acting as a reference for gaze-point detection in a HMIincluding gaze-based communication; in this case, the gaze point may bedetermined by intersecting the detected optic axis of an eye with theimage plane of the display 310 and correcting for the off-axis positionof the fovea in the retina using per se known techniques for this. Thepersonal computer system 300 further comprises an imaging device 320,which in this embodiment comprises one camera 321 and one pulsed lightsource 322 synchronized with the camera 321. Depending on the intendeduse conditions, alternative embodiment the imaging device 320 mayinclude more than one camera and more than one light source, but may aswell lack a light source altogether. The momentary power of both thecamera and the light source varies with operational parameters such asthe sampling and illumination pulse frequency, the intensity and solidangle of illumination, the image resolution and image size, so that apower-saving mode, in particular a ready mode or idle mode, may beachieved by modifying one or more of these parameters. The display 310and the imaging device 320 may be separate free-standing devices asshown on the drawing, or may form one multi-purpose unit. Alternatively,either or both may be embodied as head-mounted devices; this isparticularly advantageous in connection with a hands-free HMI of thetype outlined above.

The personal computer system 300 further comprises input means 330including a dedicated input means 331 (symbolically shown as an “off”button) for entering eye-tracking control data and a general-purposeinput means 332 (symbolically shown as a mouse). Further, the system 300includes a presence sensor 340 (shown as an optical sensor) for sensingthe presence of a viewer or, possibly, a viewer's eye, as well as anidentifying means 350, such as a biometric sensor (shown as a linescanner for fingerprints). In the figure, the peripherals discussed sofar are shown connected to a central unit 360, possibly including aprocessor (not shown), and may be embodied as physically separatecomponents or as integral parts of the central unit 360. In thisembodiment, the imaging device 320 supplies its output data to thecentral unit 360, which is responsible for executing a program (e.g., adesktop environment or application software) providing a user interfacewith which the user interacts. In portable computers and smart phones,the peripherals are commonly embodied within a common housing.

The configuration that FIG. 3 illustrates relates to a solution with arelatively low degree of hardware integration. A possible alternativehereto may be obtained by utilizing the camera 321 as a presence sensor,so that no dedicated component is required to detect user presence. Thismay be achieved with negligible inconvenience, since presence detectionis most relevant in the idle or ready mode, when the camera 321 istypically operated at reduced frame rate and/or resolution. Also, theidentifying means 350 may be integrated in the imaging device 320, e.g.,in the form of face-recognition or iris-recognition identifying means.If the imaging device 320 includes a light source 322, it isadvantageous to carry out the identification on the basis of facerecognition, as this process may be rendered more accurate by uniformillumination provided by the light source 322 rather than relying onambient light conditions.

Further, the viewer presence detector may be embodied as a proximitysensor arranged in a touch input device, such as the mouse 332 or thebutton 331 in FIG. 3. This makes it possible to predict input of newdata and put the eye tracker 320 in ready mode so as to let the delay,which is associated with this mode change, elapse at an earlier pointthan after the first input data arrive.

It will be appreciated that further integration of several functionsinto one hardware unit is possible, as is distribution of onefunctionality over several collaborating hardware units.

As shown in FIG. 1, transitions from any mode to any other mode areenabled. In this embodiment, the mode transitions are triggered bysignals provided by the presence sensor 340, a “off” button 331 forentering eye-tracking control data (forcing of the eye-trackingequipment into idle mode) and general-purpose input means 332 forentering input data other than eye-tracking data and eye-trackingcontrol data. The switching between modes may proceed as indicated inTable 2.

TABLE 2 Mode transitions From/ To Trigger condition S1 R→A Thegeneral-purpose input means 332 receive data. S2 A→R The general-purposeinput means 332 have not been used for a first predetermined timeinterval. S3 I→R The presence sensor 340 detects that a viewer ispresent. Alternative trigger: the imaging device 320 in low-power modedetects that a viewer is present and his or her ap- proximate gazedirection is at the visual display (wake on gaze). The approximatedetection may for instance be con- figured to detect two pupils that areseen in a direction close to the frontal direction, that is, wherein thepupils are moderately elliptic and do not differ above a given thresh-old from a circular shape. S4 R→I The presence sensor 340 detects thatno viewer is present. Alternative trigger: the presence sensor 340 hasnot de- tected presence of a viewer for a second predetermined timeinterval. S5 I→A The general-purpose input means 332 receive data.Alternative trigger: wake on gaze, as detailed above, op- tionallysupplemented by requiring that a dedicated input means receive data. S6A→I The presence sensor 340 detects that no viewer is present;alternatively, the presence sensor 340 has not detected presence of aviewer for a second predetermined time in- terval. S7 A→I The “off”button 331 is activated.

This embodiment achieves an object of the invention since transition S1,the resulting wake-up time of the system, requires less time thantransition S5.

The exemplifying embodiment shown in FIG. 3 lacks a positive dedicatedinput means. It will be appreciated that such positive dedicated inputmeans may be readily included, for instance, as a hardware button forinputting eye-tracking control data. The eye-tracking control data maybe input by depressing the hardware button. As explained earlier, thepressing functionality of the button may alternatively be reserved forinput of other input data that are not related to eye tracking, whereinthe eye-tracking control data are entered by a proximity sensor arrangedwithin the button. Clearly, such positive dedicated input means may insome embodiments replace the “off” button 331 shown in FIG. 3.

Turning to FIG. 2, it will now be discussed how the above setup can befurther developed by differentiating the active mode A into a set ofpersonalized active sub-modes A.1, A.2, A.3, each associated with aknown viewer (i.e., with a personal profile). This embodiment includesan initial identification step, wherein the viewer is identified usingthe identifying means 350 and the result is cached for the duration of awork session. Each transition S1, S5 into the active mode A, whetherfrom the ready mode R or the idle mode I, will then be replaced by atransition into the personalized sub-mode associated with the identifiedviewer, in accordance with the cached identification result. Similarly,each transition S2, S6, S7 from a personalized sub-mode into either theready mode R or the idle mode I may be carried out substantially as ifit happened from the active mode A.

Optionally, the sub-modes A.1, A.2, A.3 associated with the personalprofiles may be further refined into sub-profile modes A.1.a, A.1.breflecting different viewing conditions, e.g., wearing of eyeglasses, asdescribed above. Each actual viewing condition can be observedoptically. By using for instance the presence detector 340 or the camera321, which thereby acts as second identifying means in the sense of theclaims, the actual viewing condition may be continuously monitored for achange in sub-profile, allowing the settings in the active sub-mode tobe adjusted accordingly.

FIG. 4 schematically shows a configuration of modes affecting aneye-tracking processor in an eye-tracking enabled personal computersystem 300 of the general type shown in FIG. 3. An identifying means,which may be a fingerprint sensor 350 or a different component operableto verify a person's identity, identifies a user by matchinguser-related data to a predefined personal profile. In response to this,an eye-tracking processor (not shown) enters sub-mode A.1, which isassociated with the predefined profile and contains, inter alia,settings modifying a calibration step to be applied as part of theprocessing of eye-tracking data provided by an imaging device 320responsible for the eye tracking. In this mode A.1, the identifyingmeans (acting as second identifying means) attempts to sense an actualviewing condition of the user by analyzing user-related data, preferablyvisual data acquired by the imaging device 320. When an actual viewingcondition has been sensed, the eye-tracking processor enters one of thesub-profile modes A.1.a, A.1.b, A.1.c enabled by this configuration,namely the sub-profile mode associated with the sensed actual viewingcondition. Preferably, the identifying means is adapted to update(re-sense) an actual viewing condition at regular time intervals or whentriggered by predefined events. Then, if the updated condition differsfrom the current one, the eye-tracking processor enters a differentsub-profile mode. It is noted that the eye-tracking processor, which isnot shown explicitly in FIG. 3, may be located either in an imagingdevice 320 (e.g., as an embedded processor) or in a main (e.g., CPU) orauxiliary (e.g., graphical) processor 360 of the personal computersystem 300.

FIGS. 4 and 5 illustrates mode configurations applicable in a similareye-tracking enabled personal computer system 300 as the one referred toin the previous paragraph. Only one personal profile was defined in themode configuration of FIG. 4, so that the ability to recognize a userwas no essential feature of the identifying means. By contrast, FIG. 5illustrates a mode configuration in which there is only one sub-profilemode for each personalized active sub-mode. In practice, this impliesthat the identifying means utilized for implementing the modeconfiguration according to FIG. 5 need not be able to sense an actualviewing condition. In other words, this functionality is not anessential feature of the present invention. Similarly as in FIG. 4, theidentifying means carrying out the mode configuration illustrated inFIG. 5 may be adapted to determine, after initially recognizing theuser, whether the recognized user is still present or if he or she hasbeen replaced by a different user.

The algorithms illustrated by FIGS. 1, 2, 4 and 5 may be embodied ascomputer-executable instructions distributed and used in the form of acomputer-program product including a computer-readable medium storingsuch instructions. By way of example, computer-readable media maycomprise computer storage media and communication media. As is wellknown to a person skilled in the art, computer storage media includesboth volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer readable instructions, data structures, program modules orother data. Computer storage media includes, but is not limited to, RAM,ROM, EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices.Further, it is known to the skilled person that communication mediatypically embodies computer readable instructions, data structures,program modules or other data in a modulated data signal such as acarrier wave or other transport mechanism and includes any informationdelivery media.

Embodiments

-   1. A personal computer system (300) comprising:

a visual display (310);

an imaging device (320) comprising one or more cameras and being adaptedto provide eye-tracking data by imaging at least one eye of a viewer ofthe visual display; and

input means (330) for accepting eye-tracking control data and otherinput data,

wherein the imaging device is switchable between at least an activemode, a ready mode and an idle mode; and

the switching time from the idle mode to the active mode is longer thanthe switching time from the ready mode to the active mode.

-   2. The personal computer system of embodiment 1, wherein the active    mode and the ready mode differ with respect to an operating    frequency of the imaging device.-   3. The personal computer system of embodiment 1 or 2, wherein:

the eye-tracking data which the imaging device is configured to providein the active mode include eye position and eye orientation; and

the eye-tracking data which the imaging device is configured to providein the ready mode include eye position but not eye orientation.

-   4. The personal computer system of any of the preceding embodiments,    wherein the imaging device is configured to use active eye    illumination in the active mode; and

the imaging device is configured to disable active eye illumination inthe ready mode.

-   5. The personal computer system of any of the preceding embodiments,    wherein the eye-tracking data which the imaging device is configured    to track, in the ready mode, relate to an eye location in an image    plane but not in world coordinates.-   6. The personal computer system of any of the preceding embodiments,    wherein the imaging device is configured to activate, in the active    mode, at least one additional unit selected from the group    comprising

camera, and

light source,

wherein said at least one additional unit is disabled in the ready mode.

-   7. The personal computer system of any of the preceding embodiments,    wherein the imaging device is configured to pre-process optical data    by binning at a higher binning ratio in the ready mode than in the    active mode.-   8. The personal computer system of any of the preceding embodiments,    wherein the input means consist of:

at least one dedicated input means (331) adapted to accept eye-trackingcontrol data only; and

at least one general-purpose input means (332) adapted to accept atleast said other input data.

-   9. The personal computer system of embodiment 8, wherein the    dedicated input means forces the imaging device into active mode.-   10. The personal computer system of embodiment 9, wherein the    dedicated input means forces the imaging device into active mode by    directly triggering an interrupt.-   11. The personal computer system of embodiment 8, wherein the    dedicated input means is a hardware touch-actuated input means    configured to force the imaging device into idle mode.-   12. The personal computer system of embodiment 11, wherein:

the imaging device is selectively connected to drive power; and

the dedicated input means forces the imaging device into idle mode bydisconnecting it from said drive power.

-   13. The personal computer system of any of the preceding    embodiments, further comprising a viewer presence detector (340)    configured to provide at least one of:

a positive detection signal causing the imaging device to switch fromidle mode into ready mode;

a negative detection signal causing the imaging device to switch fromactive mode into idle mode;

a negative detection signal causing the imaging device to switch fromready mode into idle mode.

-   14. The personal computer system of embodiment 13, wherein:

the viewer presence detector (340) is a proximity sensor at a pushbutton and is arranged to sense proximity of a finger; and

the proximity sensor is configured to provide a positive detectionsignal causing the image device to switch from idle mode into readymode.

-   15. The personal computer system of any of the preceding    embodiments, wherein the imaging device is configured to use active    eye illumination in the active mode and the ready mode, the    illumination intensity in the ready mode being less than the    illumination intensity in the active mode.-   16. A method in a personal computer system comprising:

a visual display;

an imaging device comprising one or more cameras and being adapted toprovide eye-tracking data by imaging at least one eye of a viewer of thevisual display; and

input means for accepting eye-tracking control data and other inputdata,

said method comprising switching the imaging device between at least anactive mode, a ready mode and an idle mode, wherein the switching timefrom the idle mode to the active mode is longer than the switching timefrom the ready mode to the active mode.

-   17. A computer program product comprising a data carrier storing    instructions for causing a programmable computer to execute the    method of embodiment 16.

1. A personal computer system comprising: a visual display; an imagingdevice adapted to provide eye-tracking data by imaging at least one eyeof a viewer of the visual display; and first identifying means forrecognizing the viewer with reference to one of a plurality ofpredefined personal profiles, wherein the personal computer systemfurther comprises an eye-tracking processor for processing saideye-tracking data, which eye-tracking processor is selectively operablein one of a plurality of personalized active sub-modes associated withsaid personal profiles.
 2. The personal computer system of claim 1,further comprising a second identifying means adapted to sense arecognized viewer's actual viewing condition, wherein the imaging deviceis further operable in a sub-profile mode associated with said actualviewing condition.
 3. The personal computer system of claim 2, whereinthe second identifying means coincides with the first identifying means.4. The personal computer system of claim 2, wherein said actual viewingconditions differ by the viewer's wearing a viewing aid and/or agarment.
 5. The personal computer system of claim 1, wherein saidpersonalized active sub-modes differ by an eye-tracking calibrationsetting to be applied by the eye-tracking processor.
 6. The personalcomputer system of claim 1, wherein the first identifying means isadapted to update a result of recognizing a viewer and, if applicable, aresult of sensing an actual viewing condition; and, if the updatedresult is different, to cause the imaging device to operate in a modeassociated with the updated result.
 7. The personal computer system ofclaim 6, further comprising a second identifying means adapted to sensea recognized viewer's actual viewing condition, wherein the imagingdevice is further operable in a sub-profile mode associated with saidactual viewing condition, and wherein the second identifying means isadapted to update a result of sensing an actual viewing condition morefrequently than the first identifying means updates a result ofrecognizing a viewer.
 8. The personal computer system of claim 6,wherein said personalized active sub-modes differ by an eye-trackingcalibration setting to be applied by the eve-tracking processor, thepersonal computer system further comprising a processor adapted toexecute a graphical user interface including visible items appearing onthe visual display, wherein the second identifying means is adapted toreceive data from the processor indicating a location of one of saidvisible items and to derive a deviation of a detected gaze point fromsaid visible item location, for thereby updating a result of recognizinga viewer.
 9. The personal computer system of claim 8, wherein at leastone of said personal profiles includes a historic average gaze-pointdeviation with which the derived gaze-point is compared.
 10. Thepersonal computer system of claim 1, wherein the first identifying meanscoincides with the imaging device.
 11. The personal computer system ofclaim 10, wherein the imaging device includes active illumination andwherein the recognition of a viewer consists in face recognition. 13.The personal computer system of claim 12, wherein the mode-transitionindicator is one of: a symbol appearing on the visual display, annon-visual signal, an acoustic signal.
 14. A method in a personalcomputer system comprising: a visual display; an imaging device adaptedto provide eye-tracking data by imaging at least one eye of a viewer ofthe visual display, wherein the imaging device is selectively operablein one of a plurality of personalized active sub-modes; an eye-trackingprocessor for processing said eye-tracking data provided by the imagingdevice; and first identifying means for recognizing the viewer withreference to one of a plurality of predefined personal profiles, saidmethod comprising: recognizing, using the first identifying means, aviewer by comparing viewer-related data with predefined personalprofiles and selecting a best matching personal profile; and operatingthe eye-tracking processor in a sub-mode associated with said bestmatching personal profile.
 15. A computer program product comprising adata carrier storing instructions for causing a programmable computer toexecute the method of claim
 14. 16. The personal computer system ofclaim 3, wherein said actual viewing conditions differ by the viewer'swearing a viewing aid and/or a garment.
 17. The personal computer systemof claim 2, wherein said personalized active sub-modes differ by aneye-tracking calibration setting to be applied by the eye-trackingprocessor.
 18. The personal computer system of claim 3, wherein saidpersonalized active sub-modes differ by an eye-tracking calibrationsetting to be applied by the eye-tracking processor.
 19. The personalcomputer system of claim 4, wherein said personalized active sub-modesdiffer by an eye-tracking calibration setting to be applied by theeye-tracking processor.
 20. The personal computer system of claim 2,wherein the first identifying means is adapted to update a result ofrecognizing a viewer and, if applicable, a result of sensing an actualviewing condition; and, if the updated result is different, to cause theimaging device to operate in a mode associated with the updated result.